A critical look at quality in large-scale simulations

53 T he Jan.–Mar. 1998 IEEE Computational Science & Engineering presented two articles on the Department of Energy's Accelerated Scientific Computing Initiative, from two respected authors. John Gustafson from Ames Lab at Iowa State University discussed ASCI's computational problems. 1 Alex Larzelere, the past Director for Strategic Computing and Modeling for Department of Energy Defense Programs, discussed software engineering and management. 2 The content of the articles could not have been more diametrically opposed. Herein lies one of the biggest problems in large-scale simulations: management thinks that the " techy stuff " is a major annoyance, and the technical people do not understand the managerial need to keep costs and schedules under control. Both sides speak in the language of metrics: lines of code, test coverage, and dollars spent on one hand, and error and convergence on the other. But these metrics measure different things for different purposes. Discussions about lines of code tested and rates of convergence do not lead to meaningful communications. The role of management and science in simulation development must be changed. Software engineering is meant to produce software by a manufacturing paradigm, but this paradigm simply cannot deal with the scientific issues. This article examines the successes and failures of software engineering. I conclude that process does not develop software, people and their tools do. Second, software metrics are not meaningful when the software's purpose is to guarantee the world's safety in the nuclear era. Finally, the quality of simulations must be based on the quality of insights gained from the revealed science. Aristotle talked about it 2,300 years ago; it is time to listen. I propose a category-theory definition of validation. Is there a problem? This is not just an academic discussion. Problems such as the disconnect between management and technical people can lead to inaccurate or incomplete simulations. Failure has real consequences. Imagine this CNN news report: There is a disconnect between ASCI science and management. The author proposes an alternative paradigm. Simulation quality depends on the quality of insights gained, but software engineering is not the discipline to develop those insights. The author also proposes a definition for validation and differentiates it from current internal quality and verification approaches.